Interactive Image Annotation with Visual Feedback

A semi-automatic process, which support users in the task of annotating large image data sets, has been proposed recently. Images are clustered automatically according to similarity and are presented to the user as a sorted set. During the annotation process, partial annotations are used for further improvement of the clustering. This interactive annotation process has three important properties: First, the user is actively supported in the annotation process. Second, the basis for the clustering is visualized, thereby allowing users to understand what the underlying image features are capable of representing and distinguishing. Third, as the clustering is interactively improved certain categories may turn out to be ineffectual. We will discuss how an interactive annotation system may help to bridge the semantic gap by enhancing the users’ understanding of the underlying functionality and how the user and the learning system interact. 1 Interactive Image Annotation A semi-supervised learning approach for interactive image annotation has recently been presented by the authors in [1]. The semi-automatic user interface for the efficient annotation of image data sets clusters images according to similarity using different image features. Image annotation cannot be automated for the task of creating ground truth data for computer vision systems since correctness is crucial. Therefore the interaction between user and system is of tremendous importance. The interactive annotation process is displayed in Figure 1. Image data is initially clustered according to similarity using different image features in a Bagof-Features (BoF) approach. The clustered data is then presented to the user in an optimized user interface (UI) which facilitates the annotation of these images with custom categories. As the user annotates the images, partial annotations are used to learn a better clustering in a semi-supervised process. Identical and different annotations are interpreted as must-link and cannot-link constraints respectively. This allows the calculation of a weight vector wc for the BoF-feature vectors of category c ∈ C. The category specific vector wc is calculated to reduce the distances between images of the same category and to increase distances between images of category c and those of category g with g ∈ C ∧ g 6= c. Images which have not yet been annotated are assigned the weight vector of the 2 Moehrmann and Heidemann